Robot control system, robot system and program

ABSTRACT

A robot control system includes a processing unit which performs visual servoing based on a reference image and a picked-up image, a robot control unit which controls a robot based on a control signal, and a storage unit which stores the reference image and a marker. The storage unit stores, as the reference image, a reference image with marker in which the marker is set in an area of a workpiece or a hand of the robot. The processing unit generates, based on the picked-up image, a picked-up image with marker in which the marker is set in an area of the workpiece or the hand of the robot, performs visual servoing based on the reference image with marker and the picked-up image with marker, generates the control signal, and outputs the control signal to the robot control unit.

BACKGROUND

1. Technical Field

The present invention relates to a robot control system, a robot systemand a program or the like.

2. Related Art

Recently, industrial robots are increasingly introduced at productionsites in order to mechanize and automate works which are previouslycarried out by humans. However, accurate calibration is a prerequisitefor positioning robots and this creates a barrier to the introduction ofrobots.

Now, visual servoing is used as a technique for positioning a robot.Visual servoing is a technique of controlling a robot based on thedifference between a reference image (goal image, target image) and apicked-up image (present image). A type of visual servoing isadvantageous in that accuracy in calibration is not required andtherefore draws attention as a technique that reduces the barrier to theintroduction of robots.

Meanwhile, visual servoing has a downside that a robot cannot becontrolled unless the robot stands within a range that can be capturedby a camera.

Therefore, in order to control a robot in a range that cannot becaptured by one camera, techniques such as arranging plural cameras andproviding a reference image for each camera to perform visual servoingare conceivable.

As a technique for such visual servoing, a related art is described inKoichi Hashimoto, “Visual Servoing V—Feature-Based Visual Servoing,”Systems, Control and Information, Vol. 54, No. 5, pp. 206-213, 2010.

In visual servoing, depending on the image used, image processing maynot finish within an estimated period of time and sending of a controlsignal may be delayed, and when a robot is at a predetermined distanceor farther from a target position, image processing may fail and resultin inability to generate an effective control signal.

Moreover, when the use of a multipurpose robot capable of handlingvarious kinds of works is assumed, it is difficult to prepare necessaryreference images corresponding to all combinations of differentworkpieces and tools.

SUMMARY

An advantage of some aspects of the invention is to provide a robotcontrol system, a robot system and a program or the like which canperform visual servoing using an image with a marker set thereon.

Another advantage of some aspects of the invention is to provide a robotcontrol system, a robot system and a program or the like in which amarker and a mask image are set on a workpiece and a hand or the likeand visual servoing is performed using these images, thus restrainingthe cost of preparing reference images and also restraining theprocessing load in order to enable calculation of an effective controlsignal even when the workpiece is at a predetermined distance or fartherfrom the hand.

An aspect of the invention relates to a robot control system including:a processing unit which performs visual servoing based on a referenceimage that is an image representing a target state of a robot and apicked-up image that is an image of the robot picked up by an imagepickup unit; a robot control unit which controls the robot based on acontrol signal for the robot outputted from the processing unit; and astorage unit which stores information of the reference image and amarker for visual servoing. The storage unit stores, as the referenceimage, a reference image with marker in which at least one unit of themarker is set in an area of at least one of a workpiece and a hand ofthe robot. The processing unit generates, based on the picked-up image,a picked-up image with marker in which at least one unit of the markeris set in an area of at least one of the workpiece and the hand of therobot appearing in the picked-up image, performs visual servoing basedon the reference image with marker and the picked-up image with marker,generates the control signal, and outputs the control signal to therobot control unit. Another aspect of the invention relates to a programcauses computer to function as each of the above units, or acomputer-readable information storage medium storing the program.

According to the aspect of the invention, a reference image with markerin which at least one marker is set in an area of at least one of theworkpiece and the hand of the robot is stored as a reference image.Moreover, a picked-up image with marker in which at least one marker isset in an area of at least one of the workpiece and the hand of therobot appearing in the picked-up image is generated based on thepicked-up image. Thus, visual servoing can be carried out based on thereference image with marker and the picked-up image with marker.

According to one aspect of the invention, the storage unit may store amask image and store, as the reference image, a reference image withmarker and mask in which the mask image is set in an area of at leastone of the workpiece and the hand of the robot, with at least one unitof the marker set on the mask image. The processing unit may generate,based on the picked-up image, a picked-up image with marker and mask inwhich the mask image is set in the area of at least one of the workpieceand the hand of the robot appearing in the picked-up image, with atleast one unit of the marker set on the mask image, perform visualservoing based on the reference image with marker and mask and thepicked-up image with marker and mask, generate the control signal, andoutput the control signal to the robot control unit.

Therefore, visual servoing or the like can be carried out based on thereference image with marker and mask and the picked-up image with markerand mask. Thus, the use of the mask image enables the cost of preparingthe reference image to be restrained, and the use of the marker enablesfacilitation of homography computations or the like and calculation ofthe quantity of features of images, and the like.

According to one aspect of the invention, the processing unit mayperform visual servoing based on the reference image with marker andmask and the picked-up image with marker and mask, generate the controlsignal and output the control signal to the robot control unit, when adifference between a position of an end point of an arm of the robot andthe target position is determined as equal to or greater than a firstthreshold value.

Therefore, when the difference between the position of the endpoint ofthe arm of the robot and the target position is determined as equal toor greater than the first threshold value, visual servoing or the likecan be performed based on the reference image with marker and mask andthe picked-up image with marker and mask. Thus, when processing with alarge processing volume such as greatly moving the position of each partof the robot is carried out, restraining the processing load on robotcontrol system or the like is possible.

According to one aspect of the invention, the processing unit mayoutput, to the robot control unit, the control signal for shifting theend point of the arm to a predetermined position, when the differencebetween the position of the endpoint of the arm of the robot and thetarget position is determined as equal to or greater than the firstthreshold value. The robot control unit may control the robot based onthe control signal and shift the end point of the arm of the robot tothe predetermined position. When the robot is determined as situated atthe predetermined position, the processing unit may perform visualservoing based on the reference image with marker and mask and thepicked-up image with marker and mask, generate the control signal, andoutput the control signal to the robot control unit.

Therefore, when the difference between the position of the endpoint ofthe arm of the robot and the target position is determined as equal toor greater than the first threshold value, visual servoing or the likecan be carried out based on the reference image with marker and mask andthe picked-up image with marker and mask, after the end point of the armof the robot is shifted to the predetermined position. Thus, the numberof times of visual servoing being performed can be reduced andrestraining the processing load on the robot control system or the likeis possible.

According to one aspect of the invention, the storage unit may store, asthe reference image, a reference image without mask in which the maskimage is not set and the reference image with marker. The processingunit may perform visual servoing based on a picked-up image without maskthat is the picked-up image on which the mask image is not set and thereference image without mask, generate the control signal and output thecontrol signal to the robot control unit, when a difference between anend point of an arm of the robot and the target position is determinedas equal to or smaller than a second threshold value.

Therefore, when the difference between the position of the end point ofthe arm of the robot and the target position is determined as equal toor smaller than the second threshold value, visual servoing or the likecan be carried out based on the picked-up image without mask and thereference image without mask. Thus, when highly accurate control isrequired as in fine-tuning the position of each part of the robot,highly accurate visual servoing or the like can be carried out.

According to one aspect of the invention, the storage unit may store amask image and a reference image without mask that is the referenceimage on which the mask image is not set, and may store, as thereference image, a reference image with marker and mask in which themask image is set in the area of at least one of the workpiece and thehand of the robot, with at least one unit of the marker set on the maskimage. When the processing unit performs visual servoing based on apicked-up image without mask that is the picked-up image on which themask image is not set and the reference image without mask, and cannotdecide the control signal, the processing unit may generate a picked-upimage with marker and mask that is the picked-up image in which the maskimage is set in the area of at least one of the workpiece and the handof the robot appearing in the picked-up image, with at least one unit ofthe marker set on the mask image, perform visual servoing based on thereference image with marker and mask and the picked-up image with markerand mask, generate the control signal and output the control signal tothe robot control unit.

Therefore, when visual servoing is performed based on a picked-up imagewithout mask and the reference image without mask and the control signalcannot be decided, visual servoing or the like can be performed based onthe reference image with marker and mask and the picked-up image withmarker and mask. Thus, avoiding a situation where the robot becomesuncontrollable because of the failure to decide the control signal orthe like can be avoided.

According to one aspect of the invention, the processing unit may setthe marker by carrying out image shape recognition processing to atleast one of the workpiece and the hand of the robot.

Therefore, by carrying out image shape recognition processing to atleast one of the workpiece and the hand of the robot, setting the markeror the like is possible. Thus, setting the marker or the like ispossible without involving the cost of preparation in setting themarker.

According to one aspect of the invention, the storage unit may store thereference image with marker on which the marker without rotationalsymmetry is set. The processing unit may generate the picked-up imagewith marker on which the marker without rotational symmetry is set,perform visual servoing based on the reference image with marker and thepicked-up image with marker, generate the control signal and output thecontrol signal to the robot control unit.

Therefore, visual servoing or the like can be carried out based on thereference image with marker and picked-up image with marker on which themarker without rotational symmetry is set. Thus, reduction in the numberof markers to be set, reduction in the cost of setting markers and thelike are possible.

Still another aspect of the invention relates to a robot systemincluding the robot control system.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 shows an example of a system configuration according to anembodiment.

FIG. 2 shows a detailed example of the system configuration according tothe embodiment.

FIG. 3 is a flowchart illustrating position-based visual servoing.

FIGS. 4A and 4B are explanatory views of a reference image and apicked-up image.

FIG. 5 is a flowchart illustrating feature-based visual servoing.

FIGS. 6A and 6B are explanatory views of markers.

FIGS. 7A and 7B are explanatory views of mask images.

FIGS. 8A and 8B are explanatory views of visual servoing using markers.

FIGS. 9A and 9B are explanatory views of visual servoing using an imagein which markers are set on mask images.

FIG. 10 is an explanatory view of a visual servoing in a case where theend point of the arm is sufficiently away from the target position.

FIGS. 11A and 11B are explanatory views of a technique for performingvisual servoing after the robot is shifted to a predetermined position.

FIG. 12 is an explanatory view of visual servoing in a case where theend point of the arm is sufficiently close to the target position.

FIGS. 13A and 13B are explanatory views of a technique for settingmarkers by image shape recognition processing.

FIGS. 14A to 14C are explanatory views of a marker without rotationalsymmetry.

FIG. 15 is a flowchart of visual servoing using an image in whichmarkers are set on a mask image.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Hereinafter, an embodiment will be described. First, an example of asystem configuration according to this embodiment will be described.Next, a specific example of this embodiment will be described.Afterwards, details of processing in this embodiment will be describedwith reference to the flowcharts. The following embodiment is notintended to unduly limit the content of the invention described in theaccompanying claims. Not all the elements of the configuration describedin the embodiment are necessarily essential element of the invention.

1. Example of System Configuration 1.1 Example of Configuration of RobotSystem

An example of the configuration of a robot system according to thisembodiment is shown in FIG. 1. The robot system includes a robot controlsystem 10, an image pickup unit 20, and a robot 30. However, the robotsystem is not limited to the configuration of FIG. 1 and variousmodified embodiments are possible such as omitting one or some of thecomponents or adding another component.

The robot control system 10 generates a control signal based on an imageprovided from the image pickup unit 20 and controls the robot 30 basedon the control signal. Details of the robot control system will bedescribed later. A part or the whole of the functions of the robotcontrol system 10 according to this embodiment is realized, for example,by an information processing device such as a PC. However, a part or thewhole of the functions of the robot control system 10 may be realized bythe image pickup unit 20 and the robot 30 or by an electronic devicethat is different from an information processing device. Also, a part orthe whole of the functions of the robot control system 10 may berealized by a server connected to an information processing device orthe image pickup unit 20 and the robot 30 via communication.

The image pickup unit (camera) 20 picks up an image of a work space.This image pickup unit 20 includes an image pickup element, for example,a CCD or the like, and an optical system. The image pickup unit 20 canalso include a device (processor) used for image processing and thelike. The image pickup unit 20 is arranged at a position where the imagepickup unit 20 can pick up an image of a work space in which the robot30 and a workpiece 40 can be located. For example, the image pickup unit20 may be arranged directly above the workpiece 40 (a fixed camera) ormay be mounted on an arm 320, a hand 330 or the like of the robot 30 (amovable camera). The workpiece 40 refers to an object of processing bythe robot 30. As the image pickup unit 20, a single image pickup unitmay pick up an image of the whole or a part of the work space, or pluralimage pickup units may pick up an image of the whole or a part of thework space. The image pickup unit 20 then outputs picked-up imageinformation to the robot control system 10 or the like. Alternatively,since it suffices to be able to detect information about the position,attitude and the like of the workpiece 40 by using the image pickup unit20, other techniques than the acquisition of a picked-up image by theimage pickup unit 20, for example, three-dimension scan or the likeusing a laser or the like may be used. In this embodiment, the picked-upimage information is assumed to be directly outputted to the robotcontrol system 10, but is not limited to this configuration. Forexample, the image pickup unit 20 may be provided with a part of theprocessing unit of the robot control system 10. In such a case,information after image processing is performed to the picked-up imageis outputted.

The robot 30 has the arm 320 and the hand 330 and carries out processingaccording to a control signal from the robot control system 10. Therobot 30 carries out processing, such as, grasping and processing theworkpiece 40.

Here, the arm 320 refers a movable area that is a part of the robot 30and that includes at least one or more joints. An end point of the arm320 refers to an area at a distal end part of the arm. 320 that is notconnected with other areas than the hand 330 of the robot 30. The hand330 refers to a component mounted on the end point of the arm 320 forgrasping the workpiece 40 and processing the workpiece 40. The positionof the end point of the arm may be the position of the hand.

1.2 Example of Configuration of Robot Control System

Next, a detailed example of the configuration of the robot controlsystem according to this embodiment and a robot system including therobot control system is shown in FIG. 2.

The robot control system 10 includes a processing unit 110, a robotcontrol unit 120, a storage unit 130, and an interface (I/F) unit (inputunit) 140. The robot control system 10 is not limited to theconfiguration of FIG. 2 and various modified embodiments are possiblesuch as omitting one or some of the components or adding anothercomponent.

Next, processing carried out by each unit will be described.

The processing unit 110 carries out various kinds of processing based ondata from the storage unit 130 and information and the like from imagepickup unit 20 or the robot 30 received via the I/F unit 140. Thefunctions of the processing unit 110 can be realized by hardware such asvarious processors (CPU or the like) or ASIC (gate array or the like),or by a program or the like.

The processing unit 110 also includes a control signal generating unit112 and a picked-up image with marker generating unit 114. Theprocessing unit 110 is not limited to the configuration of FIG. 2 andvarious modified embodiments are possible such as omitting one or someof the components or adding another component.

The control signal generating unit 112 performs visual servoing based ona reference image and a picked-up image, generates a control signal forthe robot 30 and outputs the control signal to the robot control unit120. The operations of the picked-up image with marker generating unit114 will be described later. In this example, the control signalgenerating unit 112 and the picked-up image with marker generating unit114 are provided in the processing unit 110 of the robot control system.10, but these units are not limited to this configuration. The controlsignal generating unit 112 and the picked-up image with markergenerating unit 114 may be provided inside the image pickup unit 20, therobot 30 or the like.

The robot control unit 120 controls the robot 30 based on the controlsignal outputted from the processing unit 110.

The storage unit 130 stores a database and serves as a work area for theprocessing unit 110 or the like. The functions of the storage unit 130can be realized by a memory such as a RAM or by an HDD (hard disk drive)or the like.

The I/F unit 140 is an interface for entering an input or the like fromthe user to the robot control system 10 and for accepting informationfrom the image pickup unit 20 and the robot 30. In terms of entering aninput or the like from the user, the I/F unit 140 may include a switch,button, keyboard, mouse or the like.

As an example of a robot system including the robot control system 10, arobot system including the image pickup unit 20, the robot 30 and thelike can be employed.

The image pickup unit 20 is as described above. The robot 30 includes acontrol unit 310 in addition to the arm 320 and the hand 330. Thecontrol unit 310 accepts information from the robot control system 10and controls each part (arm 320, hand 330 or the like) of the robot 30.

1.3 Outline of Visual Servoing

Before explaining characteristics of this embodiment, an outline ofvisual servoing, a flow of position-based visual servoing and a flow offeature-based visual servoing will be described.

Visual servoing relates to a kind of servo system which measures achange in the position of a target as visual information and uses thisvisual information as feedback information to track the target. Visualservoing is roughly divided into two types, that is, position-basedvisual servoing and feature-based visual servoing, depending on inputinformation (control quantity) to the servo system. In position-basedvisual servoing, position information and attitude information of anobject is used as input information to the servo system. Infeature-based visual servoing, a feature quantity of an image is used asinput information to the servo system. There is also a position-basedand feature-based hybrid technique. Visual servoing utilized in thisinvention relates to all these techniques.

A common feature of these visual servoing techniques is that inputinformation to the servo system is found based on a reference image anda picked-up image. According to the invention, by setting a mask imagefor the reference image and picked-up image, the cost of preparing thereference image can be restrained and calculation of input informationto the servo system can be facilitated.

1.3.1 Flow of Position-Based Visual Servoing

First, FIG. 3 shows a flow of position-based visual servoing. Inposition-based visual servoing, a reference image is set first (S1).Here, the reference image is also called target image or goal image andis an image serving as a control target of visual servoing and showing atarget state of the robot 30. That is, the reference image is an imageshowing a target position and target attitude of the robot 30 or animage showing a state where the robot 30 is situated at a targetposition. The reference image needs to be prepared and stored in advancein the storage unit 130.

Next, the image pickup unit 20 picks up an image of a work space andacquires a picked-up image (S2). The picked-up image is an image pickedup by the image pickup unit 20. The picked-up image shows the presentstate of the work space. When the robot 30 and the workpiece 40 appearin the picked-up image, the picked-up image shows the present state ofthe robot 30 and the workpiece 40. A processing delay may occurdepending on the capability of the image pickup unit 20. Here, even whenthere is a processing delay, the picked-up image is handled as showingthe present state.

For example, FIG. 4A shows a specific example of a reference image RIM.FIG. 4B shows a specific example of a picked-up image PIM. In thepicked-up image PIM, a robot RB has an arm AM and hand HD (or endpointEP) facing upward, whereas in the reference image RIM, the robot RB isbending the arm AM and thus bringing the hand HD closer to a workpieceWK. Therefore, in this specific example, the robot RB is controlled tobend the arm AM of the robot RB and thus bring the hand HD closer to theworkpiece WK.

Next, a control command is generated (S3). For example, the controlcommand is generated by using homography or the like, which is a kind ofcoordinate transformation, based on a reference image and a picked-upimage. In this case, a homography matrix is found and a speed command isgenerated as a control signal for the robot 30, based on the homographymatrix.

Here, the control signal (control command) refers to a signal includinginformation for controlling the robot 30. The speed command refers to acommanding technique for providing the moving speed and rotating speedof the endpoint or the like of the arm 320 of the robot 30, asinformation for controlling each part of the robot 30.

Based on the generated control signal, whether a control quantity (herethe position and attitude of the robot 30) is converged to a targetvalue or not is determined (S4). For example, when homography is usedand a speed vector indicated by a speed command is 0, it can be assumedthat a target state of the position and attitude as the control quantityis reached. Therefore, the control quantity is determined as convergedto the target value. However, when the speed vector is not 0, thecontrol quantity is determined as not converged to the target value.

When the control quantity is determined as converged to the targetvalue, visual servoing ends. Meanwhile, when the control quantity isdetermined as not converged to the target value, the processing unit 110sends out a control command to the robot control unit 120.

In position-based visual servoing, the above processing is repeateduntil the control quantity is converged to the target value.

1.3.2 Flow of Feature-Based Visual Servoing

Next, FIG. 5 shows a flow of feature-based visual servoing. As inposition-based visual servoing, a reference image is set first (S10).Next, the image pickup unit 20 picks up an image of a work space andacquires a picked-up image (S11). When feature-based visual servoing isused, it is desirable that a feature is extracted from the referenceimage and a feature quantity is calculated, when the reference image isset. Alternatively, reference image information obtained by extractingthe feature of the reference image may be stored in the storage unit130. The extraction of the feature of the image is carried out, forexample, using corner detection, a Gaussian filter or the like.

Next, a feature of the picked-up image is extracted (S12). Preferablythe extraction of the feature of the picked-up image is carried out inadvance when the reference image is set, but may be carried out at thispoint. In the feature extraction, a feature quantity of the image isfound as input information (control quantity) to the visual servosystem. Then, based on the feature quantity of the image, the referenceimage and the picked-up image are compared with each other (S13) todetermine whether the reference image and the picked-up image coincidewith each other or not. When the images are determined as coincidentwith each other (S14), visual servoing ends. Meanwhile, when the imagesare determined as not coincident with each other (S14), a controlcommand is generated (S15) and the control command is sent out to therobot control unit 120 (S16).

In feature-based visual servoing, the above processing is repeated untilthe control quantity is converged to the target value.

1.4 Technique of This Embodiment

The embodiment as described above includes the processing unit 110 whichperforms visual servoing based on a reference image that is an imageshowing a target state of the robot 30 and a picked-up image that is animage of the robot 30 picked up by the image pickup unit 20, the robotcontrol unit 120 which controls the robot 30 based on a control signalfor the robot 30 outputted from the processing unit 110, and the storageunit 130 which stores the reference image for visual servoing andinformation of a marker. The storage unit 130 stores, as the referenceimage, a reference image with marker in which at least one marker is setin an area of at least one of the workpiece 40 and the hand 330 of therobot 30. Moreover, the processing unit 110 generates, based on thepicked-up image, a picked-up image with marker in which at least onemarker is set in an area of at least one of the workpiece 40 and thehand 330 of the robot 30 appearing in the picked-up image, then performsvisual servoing based on the reference image with marker and thepicked-up image with marker, generates a control signal and outputs thecontrol signal to the robot control unit 120.

In this embodiment, a reference image with marker in which at least onemarker is set in advance in an area of at least one of the workpiece 40and the hand 330 of the robot 30 is stored as the reference image.Moreover, based on the picked-up image, a picked-up image with marker inwhich at least one marker is set in an area of at least one of theworkpiece 40 and the hand 330 of the robot 30 appearing in the picked-upimage is generated. Thus, visual servoing can be performed based on thereference image with marker and the picked-up image with marker.

The marker (marker image) refers to an image of a letter, figure,symbol, pattern or stereoscopic shape that can be used as a mark, acombination of these, or a combination of these and color, and which isset in a partial area of the reference image or the picked-up image. Themarker also includes a physical object that can be picked up in an imageto generate a similar image to the above marker image and that can befixed to an object. For example, the marker may include a seal, sticker,label or the like. The shape, color, pattern or the like of the markeris not particularly limited. However, an image, seal or the like that iseasy to be distinguished from the other areas, for example, an image,seal or the like in a single color of red, is desirable.

The information of the marker may be image data or letter data used asthe marker, information used to generate the marker, or informationincluding these data. Moreover, such information of the marker may bedata that is temporarily stored when reading the data from an externalstorage device or the like.

As a specific example, FIG. 6A shows an image NMKIM that is a picked-upimage of a work space, with no markers MK set therein. FIG. 6B shows animage MKIM that is a picked-up image of the work space, with markers MKset therein. In this example, in FIG. 6B, the three grey circular areason each of the hand and the workpiece are markers MK. As the markers MKare set on the hand and workpiece, the hand and the workpiece are shownas a hand MKHD and a workpiece MKWK.

Setting markers refers to replacing arbitrary areas appearing in acreated image with markers when the image is generated. Possibletechniques for replacing arbitrary areas of the image with markers maybe, for example, superimposing marker images on the generated image,cutting out arbitrary areas from the image and inserting marker images,or attaching seals or the like used as markers to the workpiece and thehand and then picking up an image thereof. However, possible techniquesfor replacing arbitrary areas of the image with markers are not limitedto these techniques.

When markers are set on the areas of the workpiece and the hand, a partof the markers may be protruding from the areas of the workpiece and thehand. Moreover, as the areas of the workpiece and the hand, ranges witha predetermined area may be defined as the areas of the workpiece andthe hand as well as the portions where the workpiece and the handactually appear in the image.

Moreover, the reference image with marker refers to an image showing atarget state of the robot 30, with a marker set in the area of at leastone of the workpiece 40 and the hand 330 of the robot 30. For example,the reference image with marker is a reference image like MKIM shown inFIG. 6B.

The picked-up image with marker refers to a picked-up image of the robot30 by the image pickup unit 20, with a marker set in the area of atleast one of the workpiece 40 and the hand 330 of the robot 30. Forexample, the picked-up image with marker is a picked-up image like MKIMshown in FIG. 6B.

The picked-up image with marker is generated by the picked-up image withmarker generating unit 114 included in the processing unit 110.

Since visual servoing can be performed based on the reference image withmarker and the picked-up image with marker, as described above,calculation of a homography matrix or the like becomes easier in thecase of position-based visual servoing, and extraction of a featurequantity becomes easier in the case of feature-based visual servoing.Thus, the processing amount in calculating a control signal can bereduced, and an effective control signal can be calculated even when theworkpiece and the hand are at a predetermined distance or farther fromeach other.

The storage unit 130 may store a mask image and may also store, as thereference image, a reference image with marker and mask in which a maskimage is set in the area of at least one of the workpiece 40 and thehand 330 of the robot 30, with at least one marker set on the maskimage. The processing unit 110 may generate, based on the picked-upimage, a picked-up image with marker and mask in which a mask image isset in the area of at least one of the workpiece 40 and the hand 330 ofthe robot 30 appearing in the picked-up image, with at least one markerset on the mask image, perform visual servoing based on the referenceimage with marker and mask and the picked-up image with marker and mask,generate a control signal and output the control signal to the robotcontrol unit 120.

Thus, visual servoing or the like can be carried out based on thereference image with marker and mask and the picked-up image with markerand mask.

Here, the mask image is an image that is set for covering a partial areaof the work space. In this embodiment, the mask image is an image thatis set for the area of at least one of the workpiece 40 and the hand 330in the work space. The type of the mask image itself is not particularlylimited, but an image that is easy to be distinguished from the otherareas, for example, an image or the like in a single color of black, isdesirable.

FIG. 7A shows an image NMIM that is a picked-up image of the work space,with no mask image set therein. FIG. 7B shows an image MIM that is apicked-up image of the work space, with mask images set therein. Theareas where the hand HD of the robot RB and the workpiece WK appear inFIG. 7A are replaced with areas MHD and MWK in FIG. 7B. In this example,the areas in the single color of black covering the hand HD and theworkpiece WK in FIG. 7B are mask images. As the mask images area set inthe areas of the hand HD and the workpiece WK, the areas MHD and MWK areshown.

Setting a mask image refers to replacing an arbitrary area appearing ina created image with a mask image when the image is generated. Possibletechniques for replacement with a mask image is not particularly limitedand may include superimposing a mask image on the generated image, orcutting out an arbitrary area from the image and inserting a mask image,and the like.

The reference image with marker and mask refers to an image showing atarget state of the robot 30, with a mask image set in the area of atleast one of the workpiece 40 and the hand 330 of the robot 30 and withat least one marker set on the mask image. For example, the referenceimage with marker and mask is an image like MKMRIM shown in FIG. 9A,which is described later.

The picked-up image with marker and mask refers to a picked-up image ofthe robot 30 by the image pickup unit 20, with a mask image set in thearea of at least one of the workpiece 40 and the hand 330 of the robot30 and with at least one marker set on the mask image. For example, thepicked-up image with marker and mask is an image like MKMPIN shown inFIG. 9B, which is described later.

Since the reference image with marker and mask is used as the referenceimage, as described above, in some cases, the same reference image withmarker and mask can be used even when the workpiece and the hand aredifferent. Therefore, the cost of preparing the reference image can berestrained, compared with the case where reference images for all thecombinations of various kinds of workpieces and hands are prepared.Thus, the cost of preparing the reference image can be restrained byusing the mask images, and computations for homography or the like andcalculation of the feature quantity can be facilitated by using themarkers.

When the processing unit 110 determines that the difference between theposition of the end point of the arm 320 of the robot 30 and the targetposition is equal to or greater than a first threshold value, theprocessing unit 110 may perform visual servoing based on the referenceimage with marker and mask and the picked-up image with marker and mask,generate a control signal and output the control signal to the robotcontrol unit 120.

Here, the difference between the position of the end point of the arm320 and the target position includes not only the linear distancebetween the two points in a three-dimensional space but alsomathematically equivalent information. An example may be the distancebetween the position of the endpoint of the arm 320 and the targetposition in a two-dimensional picked-up image. A specific example is adistance L1 between a center point WCP of the workpiece WK (targetposition) and a center point HCP of the hand HD in a two-dimensionalimage, as shown in FIG. 10, which is described later. Additionally, whenonly one of the position of the end point of the arm 320 and the targetposition appears in the picked-up image, the difference between theposition of the end point of the arm 320 and the target position may beset to a predetermined value that is greater than the first thresholdvalue.

The first threshold value is a value that serves as a reference fordetermining whether the position of the end point of the arm 320 and thetarget position are sufficiently away from each other or not. As thefirst threshold value, a reference value corresponding to the value usedas the difference between the position of the end point of the arm 320and the target position is set. The first threshold value may be set inadvance or may be calculated by the processing unit 110 or the like. Forexample, the first threshold value may be a value such as a radius R1 ofa circle about the center point WCP of the workpiece WK (targetposition) in the two-dimensional image, as shown in FIG. 10, which isdescribed later.

Thus, when the difference between the position of the end point of thearm 320 and the target position is determined as equal to or greaterthan the first threshold value, the position of the end point of the arm320 and the target position can be determined as sufficiently away fromeach other. In such a case, visual servoing or the like can be carriedout based on the reference image with marker and mask and the picked-upimage with marker and mask. When the position of the end point of thearm 320 and the target position are sufficiently away from each other,the position of each part of the robot is often greatly moved, oftenresulting in a large processing volume. Since visual servoing based onthe reference image with marker and mask and the picked-up image withmarker and mask is advantageous in that the processing volume can berestrained, as described above, restraining the processing load byperforming such processing is effective.

Moreover, when the processing unit 110 determines that the differencebetween the position of the end point of the arm 320 of the robot 30 andthe target position is equal to or greater than the first thresholdvalue, the processing unit 110 may output a control signal for shiftingthe endpoint of the arm 320 to a predetermined position, to the robotcontrol unit 120. The robot control unit 120 may control the robot 30based on the control signal and shift the end point of the arm 320 ofthe robot 30 to the predetermined position. Furthermore, when theprocessing unit 110 determines that the robot 30 is situated at thepredetermined position, the processing unit 110 may perform visualservoing based on the reference image with marker and mask and thepicked-up image with marker and mask, generate a control signal andoutput the control signal to the robot control unit 120.

Here, the predetermined position refers to a position where the robot oreach part of the robot can be shifted without performing visualservoing. For example, the predetermined position may be an initialposition of the end point of the arm 320 of the robot 30 or may be apreset position other than the initial position.

Thus, when the difference between the position of the end point of thearm 320 of the robot 30 and the target position is determined as equalto or greater than the first threshold value, visual servoing or thelike can be carried out based on the reference image with marker andmask and the picked-up image with marker and mask after the end point ofthe arm 320 of the robot 30 is shifted to the predetermined position.

According to this technique, since the robot 30 can be shifted to apredetermined position without performing visual servoing, the number oftimes of visual servoing being performed can be reduced in some cases.Therefore, the processing load on the robot control system can berestrained.

The storage unit 130 may also store, as the reference image, a referenceimage without mask in which no mask image is set, and a reference imagewith marker. When the processing unit 110 determines that the differencebetween the end point of the arm 320 of the robot 30 and the targetposition is equal to or smaller than a second threshold value, theprocessing unit 110 may perform visual servoing based on a picked-upimage without mask that is a picked-up image in which no mask image isset and the reference image without mask, generate a control signal andoutput the control signal to the robot control unit 120.

Here, the reference image without mask refers to an image showing atarget state of the robot 30, with no mask image set therein. Forexample, the reference image without mask is a reference image like RIMshown in FIG. 4A. Whether a marker is set or not does not matter to thereference image without mask. For example, a reference image with markerMKRIM of FIG. 8A, which is described later, is also a reference imagewithout mask at the same time.

The picked-up image without mask refers to a picked-up image of therobot 30 by the image pickup unit 20, with no mask set therein. Forexample, the picked-up image without mask is a picked-up image like PIMshown in FIG. 4B. Whether a marker is set or not does not matter to thepicked-up image without mask. For example, a picked-up image with markerMKPIM of FIG. 8B, which is described later, is also a picked-up imagewithout mask at the same time.

Moreover, the second threshold value is a value that serves as areference for determining whether the position of the end point of thearm 320 and the target position are sufficiently close to each other ornot. As the second threshold value, a reference value corresponding tothe value used as the difference between the position of the end pointof the arm 320 and the target position is set. The second thresholdvalue may be set in advance or may be calculated by the processing unit110 or the like. The second threshold value may be the same value as thefirst threshold value. For example, the second threshold value may be avalue such as a radius R2 of a circle about a center point WCP of theworkpiece WK (target position) in a two-dimensional image, as shown inFIG. 12, which is described later.

Thus, when the difference between the position of the end point of thearm 320 and the target position is determined as equal to or smallerthan the second threshold value, the position of the end point of thearm 320 and the target position can be determined as sufficiently closeto each other. In such a case, visual servoing or the like can becarried out based on the reference image without mask and the picked-upimage without mask. When the position of the end point of the arm 320and the target position are sufficiently close to each other, theposition of each part of the robot is often finely tuned, oftenrequiring highly accurate control. Since visual servoing based on thereference image without mask and the picked-up image without mask isadvantageous in that detailed information of the size, shape and thelike of the workpiece and the hand can be recognized amply and thereforehighly accurate control can be performed, though a large processingvolume is required. Therefore, such processing is effective.

When visual servoing is performed based on the reference image withoutmask and the picked-up image without mask, a control signal cannot bedecided in some cases because of an extremely large processing load.

Thus, the storage unit 130 may store a mask image and a reference imagewithout mask and may also store, as the reference image, a referenceimage with marker and mask. The processing unit 110 may perform visualservoing based on a picked-up image without mask and the reference imagewithout mask. When a control signal cannot be decided, the processingunit 110 may generate a picked-up image with marker and mask, performvisual servoing based on the reference image with marker and mask andthe picked-up image with marker and mask, generate control signal andoutput the control signal to the robot control unit 120.

Therefore, when visual servoing is performed based on the picked-upimage without mask and the reference image without mask and a controlsignal cannot be decided, visual servoing or the like can be performedbased on the picked-up image with marker and mask and the referenceimage with marker and mask. Thus, it is possible to avoid a situationwhere the robot 30 becomes uncontrollable because of the inability todecide a control signal, or the like.

The processing unit 110 may set a marker for at least one of theworkpiece 40 and the hand 330 of the robot 30 by performing image shaperecognition processing.

Here, image shape recognition processing refers to a technique used torecognize the shape of an object in an image, and for example, refers tooptical flow analysis as shown in FIG. 13A, which is described later.

Thus, it is possible to set a marker or the like by performing imageshape recognition processing to the area of at least one of theworkpiece 40 and the hand 330 of the robot 30. When performing imageshape recognition processing, no preparations are needed such asattaching a marker such as a seal to the hand or the like. Therefore, itis possible to set a marker or the like without involving the cost ofpreparation when setting the marker.

The storage unit 130 may also store reference image with marker in whicha marker without rotational symmetry is set. The processing unit 110 maygenerate a picked-up image with marker in which a marker withoutrotational symmetry is set, perform visual servoing based on thereference image with marker and the picked-up image with marker,generate a control signal and output the control signal to the robotcontrol unit 120.

Thus, visual servoing or the like can be performed based on thereference image with marker and the picked-up image with marker in whicha marker without rotational symmetry is set.

Here, a marker having rotational symmetry refers to a marker whichcoincides with an original marker when the marker is rotated by one turnon two-dimensional coordinates. Therefore, if a marker having rotationalsymmetry is to be used to determine the direction or the like of thehand of the robot and the workpiece on which the marker is set, pluralmarkers need to be used to determine the direction or the like based onthe positional relation of the plural markers.

Meanwhile, a marker without rotational symmetry refers to a marker whichcoincides with an original marker only on completion of one turn whenthe marker is rotated on two-dimensional coordinates. If a markerwithout rotational symmetry is to be used to determine the direction orthe like of the hand of the robot and the workpiece on which the markeris set, it suffices to determine the direction of the one marker andthere is no need to prepare plural markers.

Therefore, the number of markers to be set can be reduced and the costof setting markers or the like can be reduced.

The robot control system or the like according to this embodiment may berealized by a program. In such a case, a processor such as a CPUexecutes the program, thus realizing the robot control system or thelike according to this embodiment. Specifically, the program stored inan information storage medium is read out, and the processor such as aCPU executes the program that is read out. Here, the information storagemedium (computer-readable medium) is for storing programs, data and thelike. The functions of the information storage medium can be realized byan optical disc (DVD, CD or the like), HDD (hard disk drive), memory(card-type memory, ROM or the like) and so on. The processor such as aCPU performs various kinds of processing of this embodiment, based onthe program (data) stored in the information storage medium. That is, inthe information storage medium, a program for causing a computer (devicehaving an operating unit, a processing unit, a storage unit and anoutput unit) to function as each unit described in this embodiment(program for causing a computer to execute processing in each unit) isstored.

2. Specific Examples

Hereinafter, specific examples of this embodiment will be described withreference to FIGS. 8A to 14C.

First, a technique of visual servoing based on a reference image withmarker and a picked-up image with marker will be described withreference to FIGS. 8A and 8B. FIG. 8A shows a reference image withmarker MKRIM. FIG. 8B shows a picked-up image with marker MKPIM.

The reference image with marker MKRIM is prepared in advance. In thiscase, after the robot RB and each part of the robot RB are arranged atexpected positions, an image of the work space is picked up by the imagepickup unit, thus generating an image. Then, a marker image MK issuperimposed on the areas of the workpiece and the hand appearing in thegenerated image and a workpiece MKWK and a hand MKHD are shown. Thus,the reference image with marker MKRIM is generated.

Meanwhile, the picked-up image with marker MKPIM is generated bysuperimposing a marker image MK on an image picked up by the imagepickup unit when visual servoing is performed actually. The techniqueused for superimposing the marker image MK is similar to the case of thereference image with marker MKRIM.

In this example, the hand MKHD in the reference image with marker MKRIMand the hand MKHD in the picked-up image with marker MKPIM are differentin position. Therefore, a control signal for bringing the hand MKHD inthe picked-up image with marker MKPIM closer to the position of the handMKHD in the reference image with marker MKRIM is calculated based onhomography or the like.

Next, a technique of visual servoing based on the a reference image withmarker and mask and a picked-up image with marker and mask will bedescribed with reference to FIGS. 9A and 9B. FIG. 9A shows a referenceimage with marker and mask MKMRIM. FIG. 9B shows a picked-up image withmarker and mask MKMPIM.

The reference image with marker and mask MKMRIM is prepared in advance,for example. In this case, after the robot RB and each part of the robotRB are arranged at expected positions, an image of the work space ispicked up by the image pickup unit, thus generating an image. Then, amask image is superimposed on the areas of the workpiece and the handappearing in the generated image, and a marker image MK is furthersuperimposed on the mask image to show a workpiece MKMWK and a handMKMHD. Thus, the reference image with marker MKMRIM is generated.

Meanwhile, the picked-up image with marker and mask MKMPIM is generatedby superimposing a mask image on an image picked up by the image pickupunit when visual servoing is performed actually, and furthersuperimposing a marker image MK on the mask image. The technique usedfor superimposing the mask image and marker image MK is similar to thecase of the reference image with marker MKRIM.

The flow of the subsequent processing is similar to FIGS. 8A and 8B.

Next, visual servoing in a case where the end point of the arm and thetarget position are sufficiently away from each other will be describedwith reference to FIG. 10. FIG. 10 shows a case where the center pointWCP of the workpiece WK coincides with the target position.

In this example, when the distance L1 between the center point WCP(target position) of the workpiece WK appearing in the picked-up imageand the center point HCP of the hand HD is equal to or greater than theradius R1 of the circle about WCP on a two-dimensional picked-up image,the end point of the arm is determined as sufficiently away from thetarget position. When L1 is smaller than R1, the end point of the arm isdetermined as not sufficiently away from the target position.

In the case of FIG. 10, since L1 is equal to or greater than R1, theendpoint of the arm is determined as sufficiently away from the targetposition. A mask image is set for the workpiece WK and the hand HD and amarker image is set further on the mask image, thus generating apicked-up image with marker and mask. Then, visual servoing is performedusing a reference image with marker and mask and the picked-up imagewith marker and mask.

Next, a technique of visual servoing after the robot is shifted to apredetermined position will be described with reference to FIGS. 11A and11B.

First, FIG. 11A shows a technique of shifting the robot RB and each partof the robot RB to the target position by performing visual servoingplural times. In this case, a range that can be covered by a first imagepickup unit is defined as VA1, a range that can be covered by a secondimage pickup unit is defined as VA2, and a range that can be covered bya third image pickup unit is defined as VA3. In this example, since thehand HD of the robot RB appears in VA1, the first visual servoing isperformed in VA1 and the hand HD is shifted to a target position in VA1.Next, visual servoing is performed sequentially in VA2 and VA3, and thehand HD is shifted to a target position in VA2 and then to a targetposition in VA3, which is the final target position.

In the case of FIG. 11A, the hand HD cannot be shifted directly to thefinal target position by the first visual servoing and therefore visualservoing with a large processing load must be repeated plural times.Also, in VA1 and VA2, the robot RB and each part of the robot RB neednot be shifted strictly to the target position and it suffices to shiftthe robot RB and each part of the robot RB accurately to the targetposition in VA3, which is the final target position. Moreover, as longas visual servoing can be carried out within VA3, the robot RB and eachpart of the robot RB can be shifted to the final target position fromany position in VA3.

Thus, a technique of performing visual servoing after shifting the robotRB and each part of the robot RB to a predetermined position that iscloser to the target position than the present position of the robot RBand each part of the robot RB is, can be considered. For example, in theexample of FIG. 11A, when the robot RB and each part of the robot RB areshifted to a predetermined position in VA2, performing visual servoingtwice enables the robot RB and each part of the robot RB to reach thefinal target position. When the robot RB and each part of the robot RBare shifted to a predetermined position in VA3, performing visualservoing once enables the robot RB and each part of the robot RB toreach the final target position.

FIG. 11B shows a case where visual servoing is performed after a centerpoint HCP1 of the hand HD is shifted to HCP2 in VA3, along a trajectoryas indicated by S1. In this example, since the predetermined positionHCP2 is the center point of the hand HD when the robot RB is in theinitial attitude, the position of the hand HD can be changed withoutperforming visual servoing. This technique is not limited to the casewhere the initial attitude is used, and can also be executed when apreset attitude or position is determined as closer to the targetposition than the present position of the robot RB and each part of therobot RB is. Using the above technique, the above advantages can beachieved and the processing load on the robot control system can berestrained.

Next, visual servoing in a case where the end point of the arm issufficiently close to the target position will be described withreference to FIG. 12. FIG. 12 shows a case where the target positioncoincides with the center point WCP of the workpiece WK.

In this example, when the distance L2 between the center point WCP ofthe workpiece WK (target position) and the center point HCP of the handHD is equal to or smaller than the radius R2 of the circle about WCP ona two-dimensional picked-up image, the end point of the arm isdetermined as sufficiently close to the target position. When L2 isgreater than R2, the end point of the arm is determined as notsufficiently close to the target position.

In the case of FIG. 12, since L2 is equal to or smaller than R2, theendpoint of the arm is determined as sufficiently close to the targetposition and visual servoing is performed using a reference imagewithout mask and a picked-up image without mask.

However, in this case, when a control signal cannot be decided based onvisual servoing because of an excessively large processing load on therobot control system, a mask image is set for the workpiece WK and thehand HD and a marker image is set on the mask image, thus generating apicked-up image with marker and mask. Then, visual servoing is performedusing a reference image with marker and mask and the picked-up imagewith marker and mask.

Next, a technique of setting a marker image by image shape recognitionprocessing will be described with reference to FIGS. 13A and 13B. FIG.13A shows an image NMKIM in which no marker image is set. FIG. 13B showsan image MKIM in which a marker image is set.

In FIG. 13A, an optical flow in the case where an optical flow analysisis carried out when picking up an image of the work space is indicatedby arrows. An area OPA including the distal end of a moving bodydetected by the optical flow analysis is specified and the hand HD isregarded as included in OPA. Then, a predetermined area in OPA existingin a direction in which the moving body travels is specified as a markersetting area MKA.

A marker image is superimposed on the specified marker setting area MKA,thus generating an image with marker MKIM. An arbitrary technique can beused for arranging the marker image in the marker setting area MKA.However, it is desirable that the marker image is arranged in such a waythat the direction of the hand and the workpiece can be determined basedon the arrangement of the marker. For example, when three markers areused, arranging the markers in such a way that the three markers form aright-angled triangle when the markers are connected by straight lines,or the like can be considered.

Finally, a technique of visual servoing using a marker withoutrotational symmetry will be described with reference to FIGS. 14A to14C.

FIGS. 14A and 14B show a marker DMK1 and a marker DMK2 as examples ofthe marker without rotational symmetry. FIG. 14C shows an image withmarker MKIM in which a marker DMK without rotational symmetry is set onthe hand, showing a hand MKHD.

The subsequent flow of processing in the case of visual servoing usingthe marker without rotational symmetry is similar to the above-describedtechnique.

3. Details of Processing

Hereinafter, an example of details of the processing according to thisembodiment will be described with reference to the flowchart of FIG. 15.

First, the image pickup unit 20 picks up an image of the workspace andgenerates a picked-up image (S20). Next, the processing unit 110estimates a linear distance in a three-dimensional space that is thedifference between the position of the workpiece 40 and the position ofthe end point (which may be the hand 330) of the arm 320, based on thepicked-up image (S21).

Then, the processing unit 110 determines whether the estimated distanceis greater than a first threshold value (S22). When the estimateddistance is determined as greater than the first threshold value, theprocessing unit 110 outputs, to the robot control unit 120, a controlsignal for shifting the robot 30 or each part of the robot 30 to apredetermined position, and the robot control unit 120 controls therobot 30 according to the control signal (S23).

Meanwhile, when the processing unit 110 determines that the estimateddistance is equal to or smaller than the first threshold value, theprocessing unit 110 compares a second threshold value with the estimateddistance (S24).

When the processing unit 110 determines that the estimated distance isgreater than the second threshold value, or after the processing of S23,the processing unit 110 decides to start visual servoing based on thepicked-up image taken by the image pickup unit 20 which shows the endpoint of the arm 320 of the robot 30, of plural image pickup units(S25). Then, the processing unit 110 reads out, from the storage unit130, a reference image with marker and mask corresponding to thepicked-up image with which visual servoing is to start (S26). Moreover,the processing unit 110 reads out a marker and a mask image form thestorage unit 130, superimposes the mask image on the acquired picked-upimage, and further superimposes the marker thereon, thus generating apicked-up image with marker and mask (S27).

Meanwhile, when the processing unit 110 determines that the estimateddistance is equal to or smaller than the second threshold value, theprocessing unit 110 decides to start visual servoing based on thepicked-up image taken by the image pickup unit 20 which shows the endpoint of the arm 320 of the robot 30, of the plural image pickup units(S28). Then, the processing unit 110 reads out, from the storage unit130, a reference image without mask corresponding to the picked-up imagewith which visual servoing is to start (S29). The picked-up imageacquired by the image pickup unit 20 is used as a picked-up imagewithout mask.

After the reference image with marker and mask is read, the processingunit 110 generates a control command for the robot 30 based on thepicked-up image with marker and mask and the reference image with markerand mask (S30). However, after the reference image without mask is read,the processing unit 110 generates a control command for the robot 30based on the picked-up image without mask and the reference imagewithout mask (S30). The control command generation processing is similarto the above-described position-based visual servoing. For example, acontrol command is generated by using homography or the like.

At this point, whether a control command can be generated or not isdetermined (S31). When a control command cannot be generated, theprocessing shifts to S25. Meanwhile, when a control command can begenerated, whether the control quantity is converged to a target valueor not is determined based on the generated control command (S32). Whenthe control quantity is determined as converged to the target value,visual servoing ends. However, when the control quantity is determinedas not converged to the target value, the processing unit 110 sends outthe control command to the robot control unit 120 (S33). Then, theprocessing shifts to S20. The flow up to this point is repeated untilthe control quantity converges.

The embodiment is described above in detail. However, those skilled inthe art can readily understand that various modifications can be madewithout substantially departing from the novel features and advantagesof the invention. Therefore, all such modifications should be regardedas included in the scope of the invention. For example, a term describedtogether with a different term with a broader meaning or the samemeaning at least once in the specification or drawings can be replacedwith that different term at any point in the specification or drawings.The configurations and operations of the robot control system, the robotsystem and the program are not limited to those described in theembodiment and various modified embodiments are possible.

The entire disclosure of Japanese Patent Application Nos. 2011-109079,filed May 16, 2011 and 2011-266540, filed Dec. 6, 2011 are expresslyincorporated by reference herein.

1. A robot control system comprising: a processing unit which performsvisual servoing based on a reference image that is an image representinga target state of a robot and a picked-up image that is an image of therobot picked up by an image pickup unit; a robot control unit whichcontrols the robot based on a control signal for the robot outputtedfrom the processing unit; and a storage unit which stores information ofthe reference image and a marker for visual servoing, wherein thestorage unit stores, as the reference image, a reference image withmarker in which at least one unit of the marker is set in an area of atleast one of a workpiece and a hand of the robot, and the processingunit generates, based on the picked-up image, a picked-up image withmarker in which at least one unit of the marker is set in an area of atleast one of the workpiece and the hand of the robot appearing in thepicked-up image, performs visual servoing based on the reference imagewith marker and the picked-up image with marker, generates the controlsignal, and outputs the control signal to the robot control unit.
 2. Therobot control system according to claim 1, wherein the storage unitstores a mask image and stores, as the reference image, a referenceimage with marker and mask in which the mask image is set in an area ofat least one of the workpiece and the hand of the robot, with at leastone unit of the marker set on the mask image, and the processing unitgenerates, based on the picked-up image, a picked-up image with markerand mask in which the mask image is set in the area of at least one ofthe workpiece and the hand of the robot appearing in the picked-upimage, with at least one unit of the marker set on the mask image,performs visual servoing based on the reference image with marker andmask and the picked-up image with marker and mask, generates the controlsignal, and outputs the control signal to the robot control unit.
 3. Therobot control system according to claim 2, wherein the processing unitperforms visual servoing based on the reference image with marker andmask and the picked-up image with marker and mask, generates the controlsignal and outputs the control signal to the robot control unit, when adifference between a position of an end point of an arm of the robot andthe target position is determined as equal to or greater than a firstthreshold value.
 4. The robot control system according to claim 3,wherein the processing unit outputs, to the robot control unit, thecontrol signal for shifting the end point of the arm to a predeterminedposition, when the difference between the position of the endpoint ofthe arm of the robot and the target position is determined as equal toor greater than the first threshold value, the robot control unitcontrols the robot based on the control signal and shifts the endpointof the arm of the robot to the predetermined position, and theprocessing unit performs visual servoing based on the reference imagewith marker and mask and the picked-up image with marker and mask,generates the control signal and outputs the control signal to the robotcontrol unit, when the robot is determined as situated at thepredetermined position.
 5. The robot control system according to claim1, wherein the storage unit stores, as the reference image, a referenceimage without mask in which the mask image is not set and the referenceimage with marker, and the processing unit performs visual servoingbased on a picked-up image without mask that is the picked-up image onwhich the mask image is not set and the reference image without mask,generates the control signal and outputs the control signal to the robotcontrol unit, when a difference between an end point of an arm of therobot and the target position is determined as equal to or smaller thana second threshold value.
 6. The robot control system according to claim1, wherein the storage unit stores a mask image and a reference imagewithout mask that is the reference image on which the mask image is notset, and also stores, as the reference image, a reference image withmarker and mask in which the mask image is set in the area of at leastone of the workpiece and the hand of the robot, with at least one unitof the marker set on the mask image, and the processing unit performsvisual servoing based on a picked-up image without mask that is thepicked-up image on which the mask image is not set and the referenceimage without mask, and when the control signal cannot be decided, theprocessing unit generates a picked-up image with marker and mask that isthe picked-up image in which the mask image is set in the area of atleast one of the workpiece and the hand of the robot appearing in thepicked-up image, with at least one unit of the marker set on the maskimage, performs visual servoing based on the reference image with markerand mask and the picked-up image with marker and mask, generates thecontrol signal, and outputs the control signal to the robot controlunit.
 7. The robot control system according to claim 1, wherein theprocessing unit sets the marker by carrying out image shape recognitionprocessing to at least one of the workpiece and the hand of the robot.8. The robot control system according to claim 1, wherein the storageunit stores the reference image with marker on which the marker withoutrotational symmetry is set, and the processing unit generates thepicked-up image with marker on which the marker without rotationalsymmetry is set, performs visual servoing based on the reference imagewith marker and the picked-up image with marker, generates the controlsignal, and outputs the control signal to the robot control unit.
 9. Arobot system comprising the robot control system according to claim 1.10. A robot system comprising the robot control system according toclaim
 2. 11. A robot system comprising the robot control systemaccording to claim
 3. 12. A robot system comprising the robot controlsystem according to claim
 4. 13. A robot system comprising the robotcontrol system according to claim
 5. 14. A robot system comprising therobot control system according to claim
 6. 15. A robot system comprisingthe robot control system according to claim
 7. 16. A robot systemcomprising the robot control system according to claim
 8. 17. A programwhich causes a computer to function as: a processing unit which performsvisual servoing based on a reference image that is an image representinga target state of a robot and a picked-up image that is an image of therobot picked up by an image pickup unit; a robot control unit whichcontrols the robot based on a control signal for the robot outputtedfrom the processing unit; and a storage unit which stores information ofthe reference image and a marker for visual servoing, wherein thestorage unit stores, as the reference image, a reference image withmarker in which at least one unit of the marker is set in an area of atleast one of a workpiece and a hand of the robot, and the processingunit generates, based on the picked-up image, a picked-up image withmarker in which at least one unit of the marker is set in an area of atleast one of the workpiece and the hand of the robot appearing in thepicked-up image, performs visual servoing based on the reference imagewith marker and the picked-up image with marker, generates the controlsignal, and outputs the control signal to the robot control unit.